In this video we learn how to fill a missing value sometimes series with backfill format fill an old
slope with interpolation so and a missing values are two bigger problems and time series because typically
we do not have 4 timestamps in our index corresponding values and our features and in particular when
we read sample re in next match time series it's very often the case that we create missing values.
So for example we do not have official stock trading prices on bank holidays or on weekends and therefore
it's quite important to handle and fill missing values.
So let's start with our close data frame.
So this is still important.
So we fear the daily closing prices for the six stocks and we have also here the additional information
on the weekday and on the quarter.
And let's let's let's look at the last five rows and we can see here on the left hand side in our day
time index that we only have four rows are timestamps for business day.
So for example here we have the thirty first of December then we have fear kind of a gap slow the next
day is the fourth of January and this is because.
So the first of January is a bank holiday.
Then the second and the third Saturday and Sunday.
So week end and as we imparted to you our stock data from Yahoo Finance.
So by default we only have fewer timestamps for business days.
And let's assume that here in our data frame we want to have all day so also the weekends and the bank
holidays and therefore be creating here a day time index with the PD up date range method and the stats
should be the thirty first of December to nine and the end date should be the very last day of our stock
or close data frame.
So it's here the sixth of February to nineteen and we want to effort daily frequencies and then we started
the day time index and the variable all days.
So let's do this and let's have a look here.
So here we have a daytime index with all the days from the thirty first of December 2009 to the 6th
of February 2019.
And now in our close data frame and set of having a daytime index that does not contain bank holidays
and weekends.
We now want to f for the daytime index with all days and we can do this by re indexing the close data
frame and became positive.
Then the brackets our new daytime time index all days and then actually we override our data frame close.
So let's do this and let's have a look at our new data frame with the new date time index.
So in our new daytime and next year we have now the timestamps.
January the 1st the second and third and consequently we FBA every man missing values.
So again he January the 1st was a bank holiday.
Then he had a Saturday and Sunday.
And here we have a Saturday and Sunday and so on.
And now our task is to actually handle and fill the missing values and for the columns day and quarter
we can do this by actually reassigning here the weekday name attribute and the attribute quarter and
to override the columns.
So let's do this here and let's re inspect so here we have now also the big days for the new timestamps
here and also the quarter.
All right and now it's time to fill the missing values in the stock price data and that actually always
depends on the specific case.
What makes most sense.
So first of all either you see the fill and a method on our data frame and let's have a look here inside
the brackets.
So we've already seen here that we can pass to the values parameter a specific scalar value.
So for example we can say that for our missing values we want to have the value 0.
And in this case this does not really make sense.
And that comes actually at the pyramid parameter method to play.
And by default it's set to None.
So A.B. can define the method to use for filling holes in re index series and by passing f fairly we
can propagate the last valid observation format to next valid or we can use backfill to use the next
valid observation.
So let's first of all pass here beefier fallback for
and if we have a look here at our data frame this are the missing values.
And with the backfill method we take the next value which is not a missing value and backfill the missing
value.
So for example here for our Apple stock we show Taffy and this is three days with the backfill method.
The value of thirty point five seven and here we have three point five seven.
So we backfilled the missing values and we can also use the forward for method
and here we actually used the last valid value.
So for example here the value at the thirty first of December and we forward fitted the missing values.
So now we should have here the value of thirty point one zero and secondly here we have it.
And as I said before it always depends on the case that which method makes most sense.
And in the case of stock prices you should actually always use them the fallout for the method.
So we use here the very last valid traded price.
And this is actually the best indication of the price at the weekend.
So in our stock example we use floorboard filler but there might be other cases there.
Beckford fear is more appropriate.
And once they are happy with our method then we can actually override our data frame and safety that
changes by passing year two to the in-place power meter
and now let's finally re inspect our data frame.
So and here we have successfully replaced all missing values.
All right.
So as a summary in this case we actually used here the area index method on our close data frame and
by doing so we created a missing values.
And then we filled and handed the missing values with the fill and a method but we can also do this
directly when re indexing other data frame so also the re index method provides here for the same tools
to handle missing values.
So also here we can see the pyramid method.
And also here we can define that.
The resulting missing values should be fed with the backfill or the forward for the method.
So we can reenact them to fill the missing various and one step what we can do this in two steps.
All right.
So now let's have a look at another example.
And therefore we import here our temperature data frame
so here we have hourly temperatures for Los Angeles and New York.
And let's assume that we want to up sample our data.
So we want to change the frequency from our lead to 30 minutes and we can do this with the every sample
method.
And as new frequency we can pass 30 minutes and we want to take you that mean.
So let's do this and by doing so we are creating your missing values.
So we're creating more timestamps than we actually have here.
So here we have data for our the uh timestamps but for the 30 minute timestamps we do not have any value.
So here we have missing values.
And also here in this example we could decide that we want to use the backfill or the forward for the
method.
However in this case this continuous temperature or better data it could also make sense to interpolate
here the values.
So for all missing values we would see the interpolated value of the values that we have before and
after some of the missing value.
So here for example for 30 minutes after midnight if you use your interpolated values then I might expect
to get to the mean of eleven point seven and ten point seven which is actually eleven point two.
And we can quite simply do this with the method interpolate.
So let's do this here and we can see that we successfully have a third on missing values are the interpolated
values.
And actually the interpolate method provides quite a lot customization.
So here we can flag somebody find the method and by default it's a linear interpolation but we can also
change this year.
And finally if you want to save our changes we also have to set the in-place perimeter to true.
All right.
So with this we are finished now with this video and I hope to see you also in the next one by.